Topics in semantic association

Learning and using language requires retrieving concepts from memory in response to an ongoing stream of information. The human memory system solves this problem by using the gist of a sentence, conversation, or document to predict related concepts and disambiguate words. Two approaches to representing gist have dominated research on semantic representation: semantic networks and semantic spaces. We take a step back from these approaches, and analyze the abstract computational problem underlying the extraction and use of gist, formulating this problem in statistical terms. This analysis allows us to explore a novel approach to semantic representation, in which words are represented using a set of probabilistic topics. The topic model performs well in predicting word association, free recall, and the senses of words, and provides a foundation for developing richer statistical models of language. Learning, speaking, and understanding language all require solving a challenging computational problem: retrieving a variety of concepts from memory in response to an ongoing stream of information. The human memory system solves this problem by using the semantic context – the gist of a sentence, conversation, or document – to predict related concepts and disambiguate words. Online processing of sentences can be facilitated by predicting which concepts are likely to be relevant before they are needed. For example, if the word BANK appears in a sentence, it might become more likely that words like FEDERAL and RESERVE would also appear in that sentence, and this information could be used to initiate retrieval of the information related to these words. This preThis work was supported by a grant from the NTT Communication Sciences Laboratory. While completing this work, TLG was supported by a Stanford Graduate Fellowship, and JBT by the Paul E. Newton chair. We thank Touchstone Applied Sciences, Tom Landauer, and Darrell Laham for making the TASA corpus available, and for their thoughts on these topics. TOPICS IN SEMANTIC ASSOCIATION 2